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14DP-03 REPORT SUMMARY 


The Montana Public Employee Retirement Administration is investing 
over $11 million to develop a new pension management information system. 
Improved monitoring and documentation of project management decisions 
will help implement the system on time and within budget. 





Context 


The Montana Public Employee Retirement 
Administration (MPERA) manages retirement 
plans for 71,000 members. These members 
include active and retired city, county, school 
district, and state employees. Employers and 
their employees rely on MPERA to calculate 
retirement benefits, create and maintain 
member activity, process claims and payments, 
along with other customer service activities. 


Information systems are used to assist MPERA 
staff in these activities, but current business 
processes are paper driven, manual, and 
time intensive. These factors have impacted 
MPERA’s customer service and ability to 
provide support for Montana’s employers and 
employees. 


For these reasons, MPERA initiated the 
MPERAtiv project. This project is multifaceted 
and estimated to cost $11.7 million. The major 
project of MPERAtiv is the implementation 
of a single, complete pension system, the 
Public Employee Retirement Information 
System (PERIS), that will increase usability 
and improve the ability to provide services to 
customers. This project started in 2012, and 
the main system is expected to be implemented 
in the summer of 2015. Multiple vendors have 
been contracted for the MPERAtiv project, 
including PERIS system development. 


Since MPERA is spending considerable 
resources on the development of PERIS and 
PERIS manages vital information, this audit 
reviewed the system development processes 
to help ensure PERIS meets expectations 
as defined in the contract. If risks are not 
mitigated throughout system development, 
the system could potentially be delayed, 
experience cost overruns, or not function as 
intended. Conducting the audit while the 
system is being developed allows MPERA to 
make improvements to the process before the 
next phase of development and through the 
remainder of the project. 


Results 


Audit work showed the expectations of 
the system set out in the contract include 
important controls such as security, data 
integrity, and user access controls. Additionally, 
MPERA has been involving users in system 
development via newsletters, workshops, 
webinars, and established an employer 
advisory group. MPERA has also developed a 
plan that, if implemented, ensures employers 
impacted by the system change will be ready 
for the implementation. 


Audit work also found established processes 
to manage expectations; however, controls 


(continued on back) 
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to ensure each expectation is being met and 
documented at each stage of the process need 
to be established. By implementing these 
controls, MPERA will have better assurance 
that its expectations of the system are met. 


MPERA established a formal change 
management process; however, in order to 
expedite changes without impact to the 
project schedule, an informal process is 
being followed. This increases the risk of 
a change being implemented without all 
involved parties knowing, without executive 
approval, or without formal realization and 
documentation of the impact to the project. 
By following a formal change management 
plan, MPERA will have better assurance that 
changes only occur if impact to the project 
is understood and all required parties have 


approved the change 


Changes within the project that reduced the 
time for user acceptance testing (UAT) were 
also identified. UAT is done to ensure system 
functionality performs in business scenarios 
and to reduce risks to MPERA after the 
system is implemented. The duration of UAT 
needs to be sufficient in order for MPERA to 
complete all necessary testing to ensure the 
system performs as expected. 


Recommendation Concurrence 


Source: Agency audit response included in 
final report. 





For a complete copy of the report (14DP-03) or for further information, contact the 
Legislative Audit Division at 406-444-3122; e-mail to lad@mt.gov; or check the web site at 


http://leg.mt.gov/audit 





Report Fraud, Waste, and Abuse to the Legislative Auditor's FRAUD HOTLINE 
Call toll-free 1-800-222-4446, or e-mail ladhotline@mt.gov. 





Chapter | — Introduction 


Introduction 


The Montana Public Employee Retirement Administration (MPERA) manages 
retirement plans for public employees within the state agencies and local governments. 
The following retirement plans are administered by MPERA. 


¢ Public Employees’ Retirement System — Defined Benefit Retirement Plan 
(PERS-DBRP) 


¢ Public Employees’ Retirement System — Defined Contribution Retirement 
Plan (PERS-DCRP) 


¢ Judges’ Retirement System (JRS) 

¢ Highway Patrol Officers’ Retirement System (HPORS) 

¢ Sheriffs’ Retirement System (SRS) 

¢ Game Warden and Peace Officers’ Retirement System (GWPORS) 
¢ Municipal Police Officers’ Retirement System (MPORS) 

¢ — Firefighters’ United Retirement System (FURS) 


¢ Volunteer Firefighters’ Compensation Act (VFCA) 
¢ 457 Deferred Compensation Supplemental Retirement Plan 


These plans consist of approximately 71,000 members and roughly $5.8 billion of 
investments for public employees. Employers and their employees rely on MPERA to 
calculate retirement benefits, create and maintain member activity, process claims and 
payments, along with other customer service activities. Information systems are used 
to assist MPERA staff in these activities. 


Background 


MPERA’s current business processes to calculate benefits and manage its members are 
paper driven and manual. These processes are time intensive due to the added manual 
approvals necessary to assure accurate processing. Spreadsheets are used to assist in 
these manual calculations, including benefit calculations and retirement processing, 
but do not completely mitigate risks of error. MPERA has recognized these factors 
impact its customer service and ability to provide support for Montana’s employers and 
employees. Along with time-intensive processes, some of the systems are over 25 years 
old, making it difficult to find Information Technology (IT) staff with the knowledge 
to maintain them. For these reasons, MPERA initiated the MPERAtiv project. 


MPERAtiv is broken out into four separate projects. The major project of MPERAtiv, 
and the focus of this audit, is the implementation of a single, complete pension 
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management information system, Public Employee Retirement Information System 
(PERIS). PERIS is projected to increase functionality and improve the ability to 
provide services to Montana employees and retirees. To fund the project, the Public 
Employees’ Retirement Board approved a portion of pension administration and plan 
investment funds each fiscal year. The MPERAtiv project began in 2012 and PERIS is 
expected to be implemented in the summer of 2015. 


System Development Life Cycle 


The PERIS system development follows a general life cycle to ensure the system is 
complete and functions well when it is time to implement. A development life cycle 
facilitates increased project efficiency, mitigates risks, and organizes the work needed 
to develop the system. The general flow of a life cycle is outlined below: 

¢ Research and Analysis. Gathering information about the system and 


developing possible solutions to a problem. Specific needs and expectations 
are identified. 


¢ Design. Creating a system that meets all of the needs and expectations 
identified earlier. 


¢ — Testing. Ensuring the system operates as expected and is ready to be used. 


¢ Implementation. Making the system available for use, including training 
users before they are expected to use the system. Monitoring system 
performance after it is implemented for errors is also done. 


¢ Support and Evolution. The daily work to manage the system and keep it 
running. 


Within this life cycle, the earlier an issue or risk is identified, the less expensive and 
quicker it will be to resolve. MPERA’s fiduciary responsibility to members to properly 
manage contributions, account balances, and benefit payments compounds the need 
for this project to run on time and efficiently. The total cost to develop and implement 
the system is estimated to be $11.7 million. To date, almost $5.2 million in development 
costs have been incurred. Due to these factors, it is important that risks and issues are 
identified and remedied as early as possible during the development of PERIS. 


Audit Scope and Objective 


This audit focuses on the development of PERIS, specifically within the design phase. 
This audit was completed pre-implementation and does not provide assurance specific 
to system functionality or performance that will be implemented. The scope of our 
work focused on the processes used by MPERA to manage PERIS project development, 
rather than specifically assessing the design or other features of the system itself. 
Following implementation of PERIS, further opportunities to audit specific aspects of 
system functionality or performance could be identified. 


Adopting a specific methodology to organize each phase of the system development 
life cycle is important to ensure the design, development, and testing of the system is 
completed accurately and efficiently. Our audit assessment found MPERA adopted a 
specific methodology for the PERIS project and is following the procedures outlined 
in the methodology. Therefore, our focus was isolated to the detailed processes of 
managing system expectations through the system development life cycle. ‘This 
includes compiling and documenting all system expectations early to ensure the 
developed system meets MPERA’s needs, designing the system to address those needs, 
and testing the system prior to implementation to ensure system expectations are 
met. In the PERIS development, these expectations are referred to as commitments. 
Commitments define exactly what MPERA expects the system to do and must be 
satisfied by the final product. Audit work reviewed process controls in place to manage 
these commitments. Based on assessment work, one objective was developed. 
Determine if controls are in place to ensure business commitments are 


developed, tested, and maintained throughout the System Development Life 
Cycle. 


Methodology 


Steps taken to conclude on this objective include: 
¢ Review commitments for system controls. 


¢ Contact other states that have implemented this system to understand their 
commitment process. 


¢ Testing a sample of commitments at various points in the life cycle for 
reference in documentation. 


¢ Verify a change management plan is established. 


¢ — Review change requests for proper documentation and approvals according 
to change management plan. 


¢ — Verify approvals were documented. 


¢ — Interview MPERA staff, contractors, and employers that will use the system 
for understanding of their involvement in the project. 


¢ Survey of employers. 
Methodology also included comparing specific PERIS processes to the terms established 
in its system development contract. Additionally, we reviewed industry best practices 


which suggest implementing specific controls to help ensure system implementation 
success. 
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Audit Conclusion 


MPERA has established commitments that include system controls in areas of 
security, user profiles, data integrity, availability, and specific system functionality. 
Other suggested best practices, such as involving users in the system development, 
were also identified. We found processes to manage commitments throughout the 
system development life cycle; however, controls should be strengthened to provide 
MPERA assurance that the system meets its needs and any changes to the system are 
approved prior to implementation. 


Chapter Il - PERIS Development 


Introduction 


The Public Employee Retirement Information System (PERIS) development is 
the main focus of the MPERAtiv project. It has been in progress since 2012 and is 
currently working through the final design stages. When PERIS is implemented, it 
is expected to be the main support system for Montana Public Employee Retirement 
Administration (MPERA) staff, including functions to manage accounts for all public 
employees, calculate benefit payments, retirement and payroll processing, process 
claims and assist in customer service. The development process is further defined in 
this chapter. 


For clarification of terminology used in this report, the following table is provided. 


Table 1 
Glossary of System Development Terms 








Term 


Definition 





System 
Development Life 
Cycle (SDLC) 


The methodology to organize the processes to design, develop, and implement an 
information management system. 





Commitments 


A statement of scope that must be satisfied by the development vendor's solution. 
Each commitment states “what” single capability shall be provided and can be 
specific to a system expectation or a project expectation. 





Commitment Path 


Commitments should follow a traceable, documented path through the development 
process to ensure that all commitments are being satisfied. 





Mapping 


A commitment is tied, or mapped, to a corresponding Use Case Scenario (UCS) so 
that commitments can be tracked throughout the project. Being able to identify what 
UCS a commitment is mapped to helps ensure all commitments are satisfied. 





Use Case 
Scenario (UCS) 


A UCS can be thought of as a grouping of related commitments, which models the 
functionality of the proposed system. UCSs define the sequence of steps and the 
interaction between user actions and system functionality. 





Business Rule 


Business rules are criteria for how a system makes decisions. They are rules that 
define aspects of business and are answered true or false. 





System Testing 


Testing is done to ensure the system is functioning as expected. Various types of 
testing are completed throughout the SDLC. System testing refers to any type of 
testing done by a technical staff member, either by MPERA staff or the development 
vendor, as opposed to being testing by a system user, or non-technical staff member. 





Test Case 


Test cases are a formal specification of a set of test inputs, execution conditions, and 
expected results identified for the purpose of making an evaluation of the test system. 
These scripts instruct the tester what to do and what to expect. 





Deliverable 


Any work product produced to support the development of the system. This includes 
UCSs, test cases, training material, or project process plans. 








User Acceptance 
Testing (UAT) 





This is the last stage of testing, performed immediately before implementation. UAT 
ensures all system functionality is successfully running as expected by the user. 





Source: Compiled by the Legislative Audit Division from MPERA records. 
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Contracted Support 


The MPERA information technology (IT) staff consists of nine FTE, which is a 
relatively small number of people to undertake a large project such as PERIS. MPERA 
identified this as a risk and contracted the development of the system to a vendor that 
has experience with pension systems. Development of PERIS will involve configuring 
an already-designed system to meet Montana’ needs. There was also a need for project 
management and quality assurance as MPERA has limited resources in these areas. A 
vendor was contracted for these services as well. The duties of each vendor are outlined 
below: 
¢ Oversight Consultant. ‘The oversight consultant is responsible for providing 
project level monitoring and review, and monitoring vendors’ compliance 
with project requirements. They review key project deliverables; the project 


schedule and budget, resources, issues and risks, and assist in the contract 
process for the other vendors described below. 


¢ Pension System Development Vendor. This vendor is providing a base 
system that is already designed, plus configurations to meet Montana’s 
needs, and training and support after implementation. They are responsible 
for overall project initiation, planning, execution, project monitoring, control 
and closure. 


¢ Data Cleansing Vendor. ‘This vendor is responsible for reviewing current 
system data (detecting errors and correcting them) and migrating data from 
the old systems to PERIS. 


MPERA established a charter in the early phases of the project that outlines the roles 
and responsibilities of each party involved as well as expectations and a hierarchy of 
authority. While contractors have been enlisted to support the MPERAtiv project, 
the charter states that the executive sponsor, MPERA, is ultimately responsible for 
the success of the project. According to the charter, success of the project is creating 
happy customers, employers, and staff, as well as improving operating efficiencies and 
increasing data quality by implementing valuable information systems. Success is also 
identified as achieving all goals within scope on time while staying on budget. 


MPERA System Development Life Cycle 


There are multiple options for system development methodologies that have been 
proven to work. The development vendor for PERIS is following the IBM Rational 
Unified Process (RUP). This methodology is established within the industry and has 
been used since 2003. The vendor providing the development service has previously 
used this method to implement pension software for other states, including Colorado, 
North Dakota, and Kansas. This process is summarized in four main phases in 


Figure 1 (on page 7). 


Figure 1 
System Development Life Cycle Phases 


Phase 1 Phase 2 Phase 3 Phase 4 








- Montana's Needs 
- Define Successful 
Management 


- Install Base System - Configuration 
- Establish Test - Testing 
Systems - Implementation 


- Defining Processes 
- Tracing Commitments 

















Source: Compiled by the Legislative Audit Division from MPERA records. 





System Development Life Cycle Phases: 


1. Project Inception. This phase identifies Montana’s specific needs 
(commitments) of a pension system and defines how the project will be 
managed to ensure its success. 


2. Commitment Confirmation. In this phase, the processes the system is 
expected to support are defined. A process for tracing the commitments 
from the contract is put in place so that no commitment is missed through 
development. 


3. Initial Infrastructure Acquisition and Installation. The development vendor 
has a base system already designed, so this phase involves preparing MPERA 
and Montana networks for installing the base model and establishing test 
systems. 


4, Solution Installation. In this phase, the base system is configured for Montana 
needs and made available to end users after thorough testing. MPERA is 
planning two releases in this phase that will include configuration, system 
testing, user training and testing, and system implementation. 


a. Implementation of the pension system for MPERA staff and 
employers. 


b. Implementation of a web-based self-service portal giving employees 
the ability to access their own information. 


This audit focused on reviewing the processes in place to ensure the commitments 
stated in the contract are satisfied. The scope was narrowed to only the development 
of the base pension system for MPERA staff and employers and does not include the 
self-service portal. Since the base pension system is a large task to complete in one 
design process, it was broken in to three parts: 4A1, 4A2, and 4A3. Each part will go 
through the same design processes, but with different portions of the system. At the 
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time of the audit, the second part, 4A2, was being finished and the third part, 4A3 
was just being started. 4A1 started in September 2012 and 4A3 is expected to finish in 
March 2015. The base pension system for MPERA staff and employers, consisting of 
these three parts, is scheduled to be implemented in June 2015. 


Report Organization 


The following chapters discuss our audit work, findings, and recommendations 
regarding PERIS system development: 


¢ Chapter III provides recommendations for improving controls over the 
detailed processes used to ensure all commitments are being satisfied. 


¢ Chapter IV discusses the importance of user acceptance testing and 
recommends improvements be made to ensure user acceptance testing is 
thorough and reduces risks of system implementation. 


Chapter III — Process Controls 


Introduction 


Three areas were reviewed to obtain assurance over the Montana Public Employee 
Retirement Administration’s (MPERA) controls and are outlined in this chapter. These 
areas were chosen to ensure that system controls were included in system expectations, 
to verify processes that ensure commitments are met by the system, and to validate 
changes to commitments are being documented. The following sections discuss audit 
work conducted, conclusions, and findings related to the management of commitments 


through the development of the Public Employee Retirement Information System 
(PERIS). 


Commitment Review 


Commitments are the expectations of the system and of the vendor stated in the 
contract. Each commitment addresses a single capability of the system or a task/ 
activity expected from the vendor. MPERA staff worked with the oversight consultant 
to develop all commitments. Creating these expectations started with commitments 
already created in previous implementations of other retirement systems. From there, 
the oversight consultant worked with MPERA staff through every business process 
to add, delete and change commitments as needed. Any changes to commitments 
after being finalized in the contract will lead to potential budget, scope, and timeline 


changes, so ensuring these commitments include system controls is essential. 


Controls Included in Commitments 


To assess commitments for important controls, the final commitments were 
individually reviewed and categorized. The audit focused on specific categories 
suggested by industry standards to be in place before a system is implemented. ‘The 
specific controls are outlined below. 


Overall System Objectives: 


¢ — Availability. The amount of time the system is expected to function properly 
should be defined. Once this is established, the impact of redundancy, 
failures, and backup processing should be formulated as well. 


¢ Data Integrity. System functionality available to ensure data is complete and 
accurate should be noted in commitments. 


Specificity of System Features: 


¢ Input. Commitments should specify the input validation method so that 
only valid data and transactions are processed. Where errors are detected, 
notifications and the ability to correct them should be provided. 
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¢ Processing. Commitments should include validation checks capable of 
detecting data that has been corrupted by processing errors and should 
describe the sequence of programs and the steps to be taken in case of a 
processing failure. 


¢ Output. Commitments for output validation are determined so that accuracy 
and completeness can be verified. 


¢ Interfaces. The definition of interfaces should be defined in commitments. 


¢ Environments. The environment the system is expected to function in 
should be defined in commitments. 


Security: 


¢ — System Security. Commitments should ensure compliance with the security 
policy of the state. 


¢ Profiles (User and Security). Profiles for users and security roles should 
be defined in commitments, and developed and tested through design. 
Procedures for maintaining profiles should also be developed prior to 
implementation. 


Audit work identified 122 commitments relative to these controls, and showed. that 
important system controls were included in the commitments stated in the contract. 


User Involvement 


It is essential for users to be involved in system development. Industry standards suggest 
users be involved for better understanding of system processes and expectations and 
to increase the use of the new system when it is implemented. Users of PERIS include 
employers and MPERA staff. Plan members’ data is managed by the system and 
employers provide the retirement and payroll data for the system. City, county, school 
district, and state employers from across the state will use PERIS, making the employer 
list over 500. Having employers involved in the development process prepares them for 
the changes they will have to undergo when the new system is implemented as well as 
identifies potential issues at an earlier time. 


MPERA staff identified the following steps taken to involve employers in the 
development of PERIS. 


1. Newsletter communication. Updates on development and what is expected 
of employers in the near future. 


2. Employer Consulting Group. Advisory group of major employers working 
closely with MPERA to define the contact between employers and PERIS. 


3. Workshops/Webinars. Presentations and discussion about the new system. 


Implementation Plan. MPERA indicated there will be a process to ensure 
all employers are ready for implementation and that all employers must be 
approved before the system can be used. 


Audit staff also created a survey for employers in the Employer Consulting Group. 
This survey was conducted to gather information about the relationship between 
employers and MPERA throughout this project. The survey was sent to 21 individuals 
with 8 responses received. The responses showed overall satisfaction with MPERA’s 
process to involve the employers, with a few noting concerns with the changes that 
need to be made to implement the system on their side. Additional discussion with 
MPERA showed they were aware of these issues and have a plan to resolve them 
before the system is implemented. Further work was done to contact smaller employers 
throughout the state with the same results. Employers were aware of the new system 
and are working with a payroll and reporting vendor that MPERA has been in contact 
with about system updates. 


Other State Implementations 


Other states and organizations were contacted to gather information about their process 
to create commitments. We contacted organizations that implemented the same base 
system, utilizing the same oversight consultant and pension system developers. ‘These 
organizations successfully implemented the project and were pleased with the vendors 
work to ensure the commitments were thorough. 


Audit work verified a process that created commitments with key system controls, such 
as those listed on page 9, was in place. Audit work also identified employers, the main 
users of the system are involved and aware of the project’s progress, and MPERA has 
developed a plan to resolve any issues with employers before the system is implemented. 


Ms 


CoNcLUSION 


Montana Public Employee Retirement Administration included important 
system controls over security, user profiles, data integrity, and specific system 
features when it compiled system expectations with the oversight consultant, 
and are including employers in the development process. 


Commitment Assurance 


Once commitments are defined, they follow a path through system development 
from creation to testing. This is done to ensure the system satisfies each commitment. 
For MPERA’s established methodology, the path starts with the commitment being 
mapped to a system process. These system processes are called Use Case Scenarios 
(UCSs). Each UCS identifies the commitment(s) it satisfies and specifically how it 
satisfies each one. Test cases, specific scenarios to be carried out by a tester, are created 
to imitate each process in the UCS and verify the processes work as expected without 
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defect. Different levels of testing are executed. Since user acceptance testing is not 
scheduled until 2015, we focused on system testing. System testing is conducted by 
an Information Technology (IT) staff member as opposed to user acceptance testing, 
which is done by a system user. These levels of testing help ensure the system is ready 
to operate as expected and agreed upon in the contract at the time of implementation. 


Commitment Sample Definition 


During the audit, a sample of 100 commitments from the contract population of 
1,347 was used in testing. Twenty were chosen specifically from those identified when 
we reviewed commitments for key system controls. These 20 commitments relate to 
industry standard suggested controls such as security, user profiles, and data integrity, 
and were chosen to gain assurance that the controls were being implemented. The 
other 80 commitments were chosen at random from the population. 


Once we identified our sample, we reviewed the commitment path to gain assurance 
the commitment would be satisfied. To determine whether the commitment path 
was adhered to and ensure testing is done to prove the commitment is satisfied, three 


points in the commitment path were reviewed. 


Commitment Mapping Review 


‘The first review was conducted to determine whether the sample of commitments were 
mapped to a UCS defining how the commitment would be met. The second review 
was conducted to determine whether the UCS referenced back to the commitment 
and defined specific business rules or processes to satisfy the commitment. These two 
reviews are highlighted in Figure 2 (on page 13). 


Figure 2 
Commitment Confirmation 





Process 


Commitment 
Mapping to 
UCS 





Mapping 
Review 





Updates 
Mappings 





Final UCS 


Source: Compiled by the Legislative Audit Division from MPERA records. 





The third review was conducted to ensure the business rules or processes defined in 
the UCS are represented in a test case. The following sections discuss the results of 
these three specific points of review and present recommendations to help ensure 
commitments are met and MPERA receives a system that meets its expectations. 


The Oversight Consultant manages a tool to trace commitments and document 
mappings to UCSs. A report of these mappings was provided by the consultant. ‘This 
report was compared to the commitment sample to test the mapping process. Audit 
work identified 88 commitments that were successfully mapped to a UCS, while ten 
were mapped to a deliverable within the project that would satisfy the commitment. 
A deliverable, other than a UCS, is a documented plan or action that satisfies the 
commitment, such as a security plan or plan to provide training to MPERA staff 
before system implementation. Two sample items were not on the report provided by 
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the Oversight Consultant, meaning the commitment was not mapped to a UCS. These 
two commitments were noted as being purged and no longer needed to be mapped to 


a UCS. 


Oa 
CONCLUSION 


Montana Public Employee Retirement Administration developed and is 
following a process to ensure commitments are mapped to specific use case 
scenarios or deliverables which increases its ability to ensure the commitment 
will be satisfied. 


UCS Reference Review 


PERIS methodology states that each UCS contains a list of the commitments mapped 
to it and identifies specifically how the UCS satisfies that commitment, whether it is 
by a certain business rule the UCS follows or a process the UCS includes. This is the 
final outcome in the commitment confirmation stage, and is needed before any system 
testing can begin. 


Each UCS was reviewed to ensure there was a reference back to that commitment 
and documentation of specific business rule or process that satisfied the commitment. 
Commitments were mapped to multiple UCSs, so of 


the 88 commitments that were mapped, 181 UCSs Table 2 
UCS Reference Results 





should have references to that mapping. From the 





























181 mappings reviewed, 89 were referenced in the Reviewed 181* 
corresponding UCS. Forty-nine mappings were to a Referenced 96 
UCS that has not been completed yet, and therefore Remapped 37 
could not be reviewed. The remaining mappings Errors 9 
were exceptions discussed with MPERA staff. After Error Rate AG7% 
further review of documentation, we identified the 

aa ; . Source: Compiled by the 
majority of these specific exceptions were due to Legislative Audit 
the commitment being remapped and would not Division from MPERA 


, records. 
constitute an error. However, some of these mappings 


*49 mappings were not reviewed 


were still found to be in error and therefore, failed die t6 prjeseuaiing 


this review. 





Commitments not being referenced in the UCS are mapped in the Oversight 
Consultant’s tool means there is no listed process or business rule to satisfy that 
commitment. This stops the documented path through system development and does 
not ensure that the commitment is satisfied by PERIS. Without knowing what satisfies 
the commitment, testing for the specific rule or process to ensure that commitment 


was met could not be executed. This could result in a system that does not satisfy all 


commitments. 


While MPERA is reviewing commitment references within the UCS, they are not 
reviewing what the Oversight Consultant's tool indicates should be mapped to the UCS. 
MPERA felt the consultant was managing this process. The Oversight Consultant is 
responsible for managing the tool; however, MPERA is ultimately responsible for the 
success of the project by ensuring the system meets their commitments outlined in 
the contract. Our audit work found commitments not represented in the UCS they 
were mapped to, supporting the need for MPERA to establish a process to ensure 
commitment mapping changes are addressed and commitments are represented in the 


corresponding UCS. 
Ma 


RECOMMENDATION #1 





We recommend Montana Public Employee Retirement Administration 
establish process controls to ensure: 


A. Commitment mapping changes are documented, executed and 
accurate, and 


Mapped commitments are represented in the corresponding use case 
scenario as indicated by the Oversight Consultant tool. 


SC 


Test Case Reference Review 


The final portion of the commitment path is the presence of the business rule or 
process that satisfies a commitment represented in a test case. According to PERIS 
methodology, test cases are created to verify the processes and business rules from the 
UCS are working properly in the system. 


System test cases are created by the development vendor and then reviewed by MPERA 
staff. After review, the test cases are executed and documented by the development 
vendor. ‘The results of testing are summarized and reviewed by MPERA for final 
approval. Figure 3 (on page 16) shows the process of system testing. ‘The system testing 
step that was audited is highlighted in red: documented system test cases. 
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Figure 3 
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Source: Compiled by the Legislative Audit Division from MPERA records. 





To conduct our review of test cases, we searched for test cases corresponding to the 


89 references found in UCSs. Errors in this review were instances of a test case not 


including the business rule or process that was listed to satisfy a commitment. Our 


review found 24 instances that failed the review and are considered errors. Two 


of these instances were justified by 
documentation to defer testing until a 
later date. Table 3 shows the results of 
this review. 


MPERA follows a process for approving 
testing completed by the development 
vendor; however, controls can be 
strengthened to ensure test cases include 
all business rules or processes that satisfy 


commitments in the UCS. With the 





Table 3 
Test Case Reference Results 





Reviewed 89 
Referenced 65 
Deferred 2 











Errors 22 





Error Rate 24.72% 














Source: Compiled by the Legislative Audit 
Division from MPERA records. 


complexity of the commitment path and ensuring they are in test cases, the amount of 


errors found in our sample are concerning considering our sample was small compared 


to the overall population. 


While reviewing these system test cases, we also found instances of test cases that did 
not include all of the required information (i.e. expected results, predefined conditions, 
executable steps) and test cases that were not run or marked executed, however, these 
test cases had been reviewed and approved by MPERA staff. According to MPERA 
staff, this testing was done at the end of the build, so time was limited to update and 
complete the documentation of testing. MPERA staff indicated they are not concerned 
about these test cases being incomplete because of the expectation that later testing 
will catch anything not tested earlier. 


If testing is not thorough and inclusive of all processes and business rules that satisfy 
commitments, there are two concerns: 1) issues could be missed, and 2) system 
functionality required in the commitments may be left out. Both of these effects may 
go unnoticed, or be uncovered in a later phases of the project when it is more costly to 
fix. This puts more pressure on user acceptance testing (the final layer) to uncover and 
resolve issues if earlier layers of testing are not thorough. By strengthening its controls 
over the test case approval process, MPERA would have increased assurance that the 
system they receive meets expectations. 


RO 


RECOMMENDATION #2 


We recommend Montana Public Employee Retirement Administration 
strengthen controls over its test case approval process to ensure: 


A. Test case documentation is compete, and 


B. Test cases address all business rules and processes from the use case 
scenario that satisfy commitments. 


To 


Change Management 


Changes are expected throughout the development process. Projects usually span a 
lengthy timeline, and business goals and initiatives may change throughout. Changes 
may also be realized when ideas from the start of the project start to be implemented 
later on. What was a good thought may not work with other parts of the project, 
or a better way to do something may be discovered. To save time or money on the 
project, the work to implement the change is weighed against the benefits. For these 
reasons, a good development methodology will include a detailed process for change 
management. Managing these changes with a defined process is important to ensure 
that the likelihood of unauthorized alteration is minimized as well. 
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Change management follows a path from creation to implementation. First, the need 
for a change is identified. This need is then documented in a change request form if 
there is a potential change to time, budget, scope, or quality of product. MPERA 
staff and the development vendor both assess the impact of the change. If there is no 
impact, a lower level of approval from managers is required, but if there is an impact 
identified, then the MPERA steering committee approval is required. If the impact is 
determined to be of a material factor, the change must be approved by the MPERA 
Board. ‘These steps can be seen in the figure below. 


Figure 4 
MPERA Change Management Process 
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Source: Compiled by the Legislative Audit Division from MPERA records. 





MPERAs Change Management Plan 


The management of changes to commitments is important to ensure MPERA receives 
the system it requests in the identified timeframe. Our audit work identified a change 
management plan established early on in the project. We reviewed key points of 
the plan to ensure it aligned with industry standards and then reviewed documents 
supporting the process to verify the plan was being followed. 


The PERIS change management plan includes: 
¢ — Clearly defined and documented procedure known by all parties. 


¢ Formal approval by management after impact assessment (change to time, 
budget, scope, or quality of product). 


¢ Change implemented after formal process is complete. 


The plan is clearly defined and identifies authorities and roles throughout change 
management also. Audit work used the plan to verify processes were followed for every 


change to the PERIS project. 


Required Formal Review of Change Requests 


While reviewing change management processes and ensuring any changes were going 
through a documented process, eleven commitments that had been purged were 
identified. Purged commitments are a change to scope and should trigger the change 
management process, so we reviewed documentation created to support these changes 
known as change request forms (CRFs). Three of the commitments were purged 
upon the acceptance of the proposal from the development vendor, before a change 
management process had been established. This was noted in documentation reviewed 
by audit staff. The remaining eight purged commitments should have followed the 
established change management process; however, audit work showed that four of 
the eight purged commitments were found in a CRF with proper approvals and the 
remaining four commitments were noted in a decision log for the project, without 


formal documentation or approval. 


We also reviewed all CRFs to determine whether appropriate documentation approval 
occurred prior to the change being implemented. While reviewing CRFs, audit staff 
found twelve change requests, with four of those being finalized. Of those four, one 
was cancelled and three were complete. Of the complete CRFs, one was found without 
all required approval signatures and one was missing documentation of the steering 
committee approval for affecting time, budget, scope or quality of product because it 
was determined that impact to scope would not be assessed at that time. Additionally, 
there was no documentation of materiality of the changes’ impact and consideration if 
they should be escalated to the Board. 
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The goal of effective change management processes is to minimize the risk of 
unauthorized changes and enforce a formal process to assess impact of the change. 
Commitments being purged without documentation increases the risk that the change 
could have been implemented without all involved parties knowing, without executive 
approval, or without formal realization and documentation of the impact to the project. 


MPERA staff indicated they are reviewing and updated the change management 
process because at this point changes have been minimal, and as the project gets further 
in to design, more changes are occurring and process improvements are necessary. 
Audit work found an informal process that did not document impact assessments in 
some cases, so the materiality of the issue may not be identified, and therefore may not 
be communicated to the Board. A current example of how this is affecting the project 
is starting to take place. The development vendor is requesting just over $100,000 
additional funding to cover potential changes to the project. Purged commitments 
could offset these requests as they are a decrease in the scope of the project. However, 
there was no formal documentation completed to assess the impact to scope when 
the changes were implemented. Now MPERA staff are reviewing all of the change 
requests to determine the impact to the project. 


While the four purged commitments without CRFs were documented on the decision 
log for the PERIS project, MPERA was using an informal process for changes to 
maintain vendor relations and not impact the project schedule. The informal process 
refers to staff using the decision log to “hold” change details so that the change can be 
implemented faster and they are not waiting for paperwork of the formal process to 
be finished. This informal process also meant that steering committee approvals and 
materiality decisions were not being documented in meeting minutes. By following its 
formal change management plan, MPERA will have better assurance that commitment 
changes only occur if impact to the project is understood and all required parties have 


approved the change. 
(ea 


RECOMMENDATION #3 





We recommend Montana Public Employee Retirement Administration only 
initiate changes after they have gone through its formal change management 
process including: 


A. Montana Public Employee Retirement Administration review of time, 
budget, scope, or quality of product and documentation supporting 
decision, 


B. Document steering committee approval, and 


C. Reviewing and documenting whether the change is material and should 
be escalated to the Montana Public Employee Retirement Administration 
Board. 


Chapter IV — User Acceptance Testing 


Introduction 


The final process in the development life cycle before implementation is user acceptance 
testing (UAT). Since UAT is not scheduled until January 2015, to this point, the 
report has discussed processes through system testing. While system testing is initial 
testing conducted by the development vendor, UAT is the last chance for the Montana 
Public Employee Retirement Administration (MPERA) to ensure the system meets 
expectations set by the contract. 


The objective of UAT is to ensure system functionality performs in business scenarios, 
therefore, testing is completed by MPERA. UAT is also done to protect MPERA from 
the following potential risks after the system is implemented. 


¢ — Legal Risk. The potential for the system to not be in compliance with law. 


¢ Time Risk. The potential for the system processes to not meet business 
deadlines. 


¢ Resource Risk. ‘The potential for issues after implementation requiring more 
resources to fix than available. 


¢ Reputation Risk. The potential for external users, employers, state, and local 
government employees to perceive a problem with the system, which could 
be interpreted as a problem with the agency. 


Industry standards state that all business scenarios, in this case Use Case Scenarios 
(UCSs), be successfully executed and documented in UAT. IF MPERA implements the 
recommendations from this report, there will be increased assurance that all business 
scenarios (UCSs) will be documented and tested thoroughly in UAT. The assurance is 
vital with a complex system that could cost almost $12 million to implement. 


PERIS UAT Methodology 

The process established in PERIS methodology for UAT follows industry standards 
and is similar to system testing, the difference being that UAT test cases are built and 
executed by MPERA staff instead of the development vendor. UAT has significant 
impact to the success of the project and mitigating the potential risks mentioned 
above. When commitments were created, the importance of UAT was identified in the 
contract, and a commitment requiring UAT timeframe be at least 10 percent of the 
total project time was created. 
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Pressure on User Acceptance Testing 


As discussed in the previous chapter, we identified system test cases that were not 
run or were incomplete. MPERA staff stated they are relying on later testing to catch 
incomplete tests in system testing. By relying on later testing, MPERA could potentially 
be increasing the work needed to be done by its own staff in UAT. Additionally, when 
reviewing commitment assurance and change management, we found changes that 
shortened the UAT timeframe. After reviewing the impact of these changes, we 
identified that the UAT schedule went from 122 days to 90 days. 


Shortening the amount of UAT on the actual system creates less time to ensure the 
system satisfies the contract. This is compounded by the fact that not all business 
rules and processes satisfying commitments are being tested in earlier layers of testing, 
creating additional work and pressure on UAT. 


With the potential impacts to UAT, there is an increased risk that user acceptance 
testing will not provide assurance the system will meet expectations defined in the 
contract. If audit recommendations are implemented through the remainder of 
the development life cycle, this risk could be decreased, however; since UAT is not 
scheduled until 2015, we cannot assure that final UAT will include all tests necessary 
to confirm all commitments. 


The duration for UAT has been decreased to the minimum required by the contract. 
While still compliant with the contract, MPERA has increased its risk of implementing 
a system that does not meet its expectations by adding work to the critical phase of 
UAT and shortening the amount of time allotted to carry out the testing. Therefore, 
we have no assurance that the amount of time scheduled is sufficient or that UAT will 
be thorough enough to confirm all commitments from the contract are being met by 
the system. 


O_O 
CONCLUSION 


The heightened importance and reduced duration of user acceptance testing 
during PERIS development increases the risk of the system not meeting 
contractual commitments or the needs of its users. 
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Dear Ms. Hunthausen: 


We appreciate the opportunity to respond to the recommendations in the information 
systems audit of the Public Employee Retirement Information System (PERIS) 
Development Life Cycle. We understand there are three recommendations. 


Recommendation #1 
We recommend the Montana Public Employee Retirement Administration establish 
process controls to ensure: 
A. Commitment mapping changes are documented, executed and accurate, and 
B. Mapped commitments are represented in the corresponding use case 
scenario as indicated by the Oversight Consultant Tool. 


Response 
We concur. 


A. Corrective action plan: Our Oversight Consultant vendor will monitor the CTS and 
report to MPERA weekly on project commitments in their Weekly Project Monitoring 
and Feedback Report. Their report will include commitment metrics, commitment 
changes and incorrectly mapped commitments. The MPERAtiv Program Manager will 
review this report weekly and work with the development vendor and our Oversight 
Consultant to resolve any discrepancies. 

Timeline for implementation of recommendation: Implemented. 


B. Corrective action plan: MPERA will include a review of the commitments tracked in 
the Commitment Tracking System (CTS) as part of the formal acceptance of each UCS. 
This review will confirm that all commitments mapped to the use case in CTS are 
addressed in the UCS and all commitments addressed in the UCS are also mapped 
correctly in CTS. Any discrepancies will be investigated and resolved prior to the 
acceptance of the use case. 

Timeline for implementation of recommendation: Implemented. 
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Recommendation #2 
We recommend the Montana Public Employee Retirement Administration 
strengthen controls over its test case approval process to ensure: 
A. Test case documentation is complete, and 
B. Test cases address all business rules and processes from the use case scenario 
that satisfy commitments. 


Response 
We concur and would like to differentiate between Unit test Case (UTC) and System Test 


Case (STC) reviews. 

A. UTC are not formal deliverables to MPERA but are artifacts from the construction 
process. Acceptance of the UTC is not tied to formal acceptance of the completion of 
construction; MPERA reviews the UTC and marks them as reviewed prior to 
construction acceptance. It was determined that an in depth review of the UTC was not 
needed as the UTC involves the testing of individual components to ensure the system 
meets the use case specifications and is ready for system testing, the next level of testing. 
Our Embedded Staff would focus more time on their primary goal of training to develop 
and support PERIS when it goes live than on reviewing UTC. 


STC documents are also not formal deliverables to MPERA. Our development vendor 
has a defined procedure for completing STC documents and strives for complete, 
thorough, quality testing, along with ensuring the STC documents completely reflects the 
test case, test steps, expected results, and actual results and the methods for documenting 
are clear (e.g. bullets, snippets). 


Our review of the STC documentation is stringent and also includes a review of the 
System Test Phase summary documentation provided by our development vendor. This 
summary documentation details their review of the system test phase and includes overall 
statistics such as number executed, passed, failed, not applicable, deferred, or not 
testable. 


Corrective action plan: We have established a test case review checklist which includes 
verification that all UTC and STC documentation is complete including all required 
information. 

Timeline for implementation of recommendation: Implemented. 


B. This recommendation is not applicable to UTC as the goal of unit testing is the testing 
of individual components to ensure the system meets the use case specifications and is 
ready for system testing. 


Corrective action plan: We have established a test case review checklist which includes 
verification that each business rule or process that satisfies a commitment applicable to 
the use case has a corresponding STC. 

Timeline for implementation of recommendation: Implemented. 
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Recommendation #3 
We recommend the Montana Public Employee Retirement Administration only 
initiate changes after they have gone through its formal change management 
process including: 
A. Montana Public Employee Retirement Administration review of time, 
budget, scope, or quality of product and documentation supporting decision, 
B. Document steering committee approval, and 
C. Reviewing and documenting whether the change is material and should be 
escalated to the Montana Public Employees Retirement Board. 


Response 
We concur. 


MPERA implemented an MPERAtiv Change Control Board (CCB) on July 15, 2014. 
The CCB reviews all project change request forms (CRF) and decision documents (DD) 
and recommends a disposition to the Project Sponsor and Executive Sponsor. 

The meeting minutes of the CCB are recorded and stored on SharePoint. 


Corrective Action Plan: MPERA will ensure the CCB reviews all CRF and DD for 
impacts to time, budget, scope or quality and determines if the change is material and 
should be escalated to the Montana Public Employees’ Retirement Board. This 
information will be recorded in the CCB meeting minutes. 

Timeline for implementation of recommendation: Implemented. 


We appreciate the time and effort expended to complete this information systems audit 
and the courtesy and consideration extended to MPERA during the audit. 


Thank you for your assistance. 


ore Schwinden 
Executive Director 


"AN EQUAL OPPORTUNITY EMPLOYER" 


